Operating systems can be found on almost any device that contains a computing system from cellular phones and video game consoles to supercomputers and web servers. A device's operating system (OS) is a collection of software that manages computer hardware resources and provides common services for user application programs. The OS typically acts as an interface between the hardware and the programs requesting input or output (I/O), CPU resources and memory allocation. When an application executes on a computer system with an operating system, the application's code is usually executed directly by the hardware and can make system calls to the OS or be interrupted by it. The portion of the OS code that interacts directly with the computer hardware and implements services for applications is typically referred to as the kernel of the OS. The portion that interfaces with the applications and users is known as the shell. The user can interact with the shell using a variety of techniques including (but not limited to) using a command line interface or a graphical user interface (GUI).
Most modern computing devices support graphical user interfaces (GUI). GUIs are typically rendered using one or more interface objects. Actions in a GUI are usually performed through direct manipulation of graphical elements such as icons. In order to facilitate interaction, the GUI can incorporate one or more interface objects referred to as interaction elements that are visual indicators of user action or intent (such as a pointer), or affordances showing places where the user may interact. The term affordance here is used to refer to the fact that the interaction element suggests actions that can be performed by the user within the GUI.
A GUI typically uses a series of interface objects to represent in a consistent manner the ways in which a user can manipulate the information presented to the user via the user interface. In the context of traditional personal computers employing a keyboard and a pointing device, the most common combination of such objects in GUIs is the Window, Icon, Menu, Pointing Device (WIMP) paradigm. The WIMP style of interaction uses a virtual input device to control the position of a pointer, most often a mouse, trackball and/or trackpad and presents information organized in windows and/or tabs and represented with icons. Available commands are listed in menus, and actions can be performed by making gestures with the pointing device.
The term user experience is generally used to describe a person's emotions about using a product, system or service. With respect to user interface design, the ease with which a user can interact with the user interface is a significant component of the user experience of a user interacting with a system that incorporates the user interface. A user interface in which task completion is difficult due to an inability to accurately convey input to the user interface can lead to negative user experience, as can a user interface that rapidly leads to fatigue.
Touch interfaces, such as touch screen displays and trackpads, enable users to interact with GUIs via two dimensional (2D) gestures (i.e. gestures that contact the touch interface). The ability of the user to directly touch an interface object displayed on a touch screen can obviate the need to display a cursor. In addition, the limited screen size of most mobile devices has created a preference for applications that occupy the entire screen instead of being contained within windows. As such, most mobile devices that incorporate touch screen displays do not implement WIMP interfaces. Instead, mobile devices utilize GUIs that incorporate icons and menus and that rely heavily upon a touch screen user interface to enable users to identify the icons and menus with which they are interacting.
Multi-touch GUIs are capable of receiving and utilizing multiple temporally overlapping touch inputs from multiple fingers, styluses, and/or other such manipulators (as opposed to inputs from a single touch, single mouse, etc.). The use of a multi-touch GUI may enable the utilization of a broader range of touch-based inputs than a single-touch input device that cannot detect or interpret multiple temporally overlapping touches. Multi-touch inputs can be obtained in a variety of different ways including (but not limited to) via touch screen displays and/or via trackpads (pointing device).
In many GUIs, scrolling and zooming interactions are performed by interacting with interface objects that permit scrolling and zooming actions. Interface objects can be nested together such that one interface object (often referred to as the parent) contains a second interface object (referred to as the child). The behavior that is permitted when a user touches an interface object or points to the interface object is typically determined by the interface object and the requested behavior is typically performed on the nearest ancestor object that is capable of the behavior, unless an intermediate ancestor object specifies that the behavior is not permitted. The zooming and/or scrolling behavior of nested interface objects can also be chained. When a parent interface object is chained to a child interface object, the parent interface object will continue zooming or scrolling when a child interface object's zooming or scrolling limit is reached.
The evolution of 2D touch interactions has led to the emergence of user interfaces that are capable of 3D interactions. A variety of machine vision techniques have been developed to perform three dimensional (3D) gesture detection using image data captured by one or more digital cameras (RGB and/or IR), or one or more 3D sensors such as time-of-flight cameras, and structured light cameras. Detected gestures can be static (i.e. a user placing her or his hand in a specific pose) or dynamic (i.e. a user transition her or his hand through a prescribed sequence of poses). Based upon changes in the pose of the human hand and/or changes in the pose of a part of the human hand over time, the image processing system can detect dynamic gestures.